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BRIEF 


This  investigation  compared  the  performance  of  a  group  of 
subjects  assessed  on  a  simulated  troubleshooting  task  and  on  the 
identical  actual  troubleshooting  task  using  real  equipment.  Sub¬ 
jects  were  14  students  in  the  experimental  training  program  for 
Electronics  Technicians ,  conducted  by  the  Navy  Training  Research 
Laboratory,  San  Diego. 

Analysis  of  results  revealed  that  the  simulated  performance 
measure  did  not  provide  a  valid  estimate  of  performance  proficiency 
on  the  actual  task.  Obtained  negative  inter-test  correlations 
indicate  that  simulated  test  r<  ,ults  would  actually  be  misleading 
in  terms  of  estimating  actual  performance  scores.  In  addition  to 
performance  score  discrepancies,  there  were  observable  differences 
in  specific  performance  piocedures  and  overall  troubleshooting 
strategy  attributable  to  the  differences  in  test  mode.  The  evidence 
strongly  suggests  caution  in  assuming  that  a  simulated  performance 
measure,  even  with  considerable  face  validity,  will  provide  a 
valid  estimate  of  actual  performance  on  a  common  task. 
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COMP ARISON  OF  PERFORMANCE  ON  ANALOGOUS  SIMULATED  AND 
ACTUAL  TROUBLESHOOTING  TASKS 

A.  Background 


This  exploratory  investigation  is  a  part  of  the  larger  research 
effort  to  develop  and  implement  a  practicable  system  for  valid 
assessment  of  the  performance  skills  required  for  job  proficiency 
in  Navy  ratings.  There  is  an  expressed  need  in  the  Navy  for  methods 
of  assessing  proficiency  on  actual  job  performance  tasks,  since 
practical  performance  skills  are  often  the  most  critical  component 
of  total  job  competence,  yet  they  are  presently  least  amenable  to 
objective  measurement.  Intellectual  aptitudes  and  knowledge  require¬ 
ments  for  any  particular  rating  may  be  reasonably  well  measured  by 
conventional  paper  and  pencil  tests,  but  valid  measures  of  profi¬ 
ciency  on  the  practical  factors  of  the  rating  are  not  generally 
available.  For  ratings  in  which  performance  chiefly  involves  admin¬ 
istrative  and  clerical  duties,  paper  and  pencil  tests  may  constitute 
a  reasonable  job  task  sample.  In  technical  ratings  requiring 
complex  practical  job  skills,  such  as  troubleshooting  in  the  elec¬ 
tronics  ratings,  proficiency  evaluation  requires  tests  with  broader 
measurement  capability.  Despite  the  enormous  amount  of  effort 
which  has  been  devoted  to  the  problem  of  assessing  troubleshooting 
proficiency,  there  remains  a  large  number  of  relatively  unsolved 
theoretical  and  practical  assessment  problems.  Ideally  a  trouble¬ 
shooting  proficiency  test  for  Navy-wide  use  should  be  portable, 
brief,  inexpensive,  simple  to  administer  and  score,  and  yet  yield 
a  valid  estimate  of  actual  troubleshooting  ability  on  the  job. 
Unfortunately,  currently  available  tests  lack  some  or  most  of  these 
desirable  features.  One  the  alternative  approaches  to  perform¬ 
ance  testing  has  been  an  attempt  to  develop  simulated  tests  which, 
by  substitution,  can  reduce  some  of  the  time,  cost,  and  technical 
requirements  of  actual  performance  tests.  The  true  value  of  such 
simulated  tests,  of  course,  depends  upon  the  extent  to  which  they 
can  provide  valid  proficiency  estimates  of  the  actual  performance 
task  being  measured. 

The  present  study,  accordingly,  is  concerned  with  evaluative 
comparisons  of  the  results  obtained  from  assessment  of  trouble¬ 
shooting  proficiency  using  a  simulated  performance  task  and  using 
the  analogous  actual  troubleshooting  performance  task.  Such 
experimental  comparisons  provide  information  prerequisite  to 
development  and  implementation  of  improved  performance  evaluation 
systems . 


B,  Experimental  Procedures 

Two  highly  analogous  tasks,  one  presenting  a  troubleshooting 
problem  in  Emulated  form  and  the  other  being  the  actual  trouble¬ 
shooting  pex'formance  task,  were  utilized  as  a  basis  for  performance 
comparisons.  The  investigation  was  more  concerned  with  determining 
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the  similarities  and  differences  of  task  performance  under  the  two 
assessment  modes  than  in  simply  establishing  the  score  validities 
of  the  simulated  task.  For  this  purpose  the  development  of  an 
identical  test  problem  was  essential  in  order  to  permit  step  by 
step  comparison  of  the  troubleshooting  approach,  logic  and  sequence. 
Individual  performance  on  each  version  of  the  task  could  then  be 
contrasted  in  terms  of  specific  performance  characteristics,  and  in 
terms  of  correlational  comparisons  of  derived  performance  scores. 

1.  The  Tasks 


a.  Simulated .  The  troubleshooting  Electronics-Trainers  for 
the  Superhetei*odyne  Receiver  (Van  Valkenburgh,  Nooger  and  Neville, 
Inc.  NAVPERS  92088-5B)  was  used  as  the  vehicle  for  the  simulated 
performance  task.  An  alternate  form  of  the  same  series  (NAVPERS 
92088-9B)  served  as  a  practice  vehicle  in  order  to  assure  that 
subjects  had  recent  familiarity  with  the  procedures  of  the  simu¬ 
lated  task.  These  instruments  are  among  the  most  commonly  found 
simulated  training  and  testing  devices  used  in  Navy  technical 
training  courses.  The  two  forms  utilize  the  same  procedural 
format  but  differ  with  regard  to  the  specific  superheterodyne 
casualty  which  is  the  basis  of  the  test  problem.  The  casualty 
problem  of  the  first  form  is  identical  to  that  used  for  the  actual 
performance  task,  and  involves  a  faulty  resistor  (R-ll)  which 
produces  only  a  low  hum  as  the  loudspeaker  output.  The  practice 
form  casualty  involved  a  faulty  tube  (V-6)  which  resulted  in  no 
loudspeaker  output,  i’hese  troubleshooting  tests  present  a  verbal 
description  of  the  problem  casualty  symptoms  which  the  examinee 

is  expected  to  correct  by  a  series  of  simulated  troubleshooting 
steps.  To  perform  each  of  these  steps,  the  examinee  successively 
selects  appropriate  test  point  checks,  measurements,  or  parts 
replacement  alternatives  from  among  the  many  possible  choices 
listed  on  the  Trainer-Tester  sheet.  For  each  selected  trouble¬ 
shooting  step,  the  examinee  must  erase  the  sil\rer  covering  material 
which  obscures  the  information  given  for  each  item.  This  informa¬ 
tion  consists  of  a  resistance,  voltage,  or  output  signal  reading, 
or  the  result,  of  making  a  specific  parts  replacement.  The  examinee 
attempts  to  solve  the  problem  by  erasing  as  few  items  as  possible 
until  he  locates  the  item  which  yields  "TC"  (trouble  corrected)  to 
indicate  that  he  has  repaired  the  casualty.  A  detailed  schematic 
diagram  of  the  equipment  is  provided  as  an  accompanying  task  aid 
to  the  testee. 

b.  Actual .  The  actual  troubleshooting  performance  task  was 
the  real  equipment  replication  of  the  troubleshooting  task  presented 
in  the  simulated  version.  Superheterodyne  radio  receivers  were 
constructed  directly  from  the  detailed  schematic  diagram  of  the 
simulated  task.  The  identical  component  casualty  (R-ll)  of  the 
simulated  task  was  then  inserted  into  these  vehicles  to  provide  an 
actual  duplication  of  the  simulated  test  vehicle  and  problem. 
Examinees  were  required  to  isolate  and  repair  this  casualty  using 
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symptom  information  and  a  detailed  schematic  diagram  identical  to 
that  of  the  simulated  task. 


2.  The  Sample 

Subjects  for  this  investigation  were  14  students  in  the 
experimental  training  program  for  Electronics  Technicians,  con¬ 
ducted  by  the  Navy  Training  Research  Laboratory,  San  Diego, 

1964  -  1966.  All  subjects  had  completed  the  Fundamentals  phase 
of  the  course  and  about  17  weeks  of  the  advanced  Equipment  phase 
(Radar  or  Communications)  at  the  time  of  the  first  testing  session. 

3.  Assessment  Procedures 


The  simulated  and  actual  performance  tests  were  administered 
to  the  trainees  in  counterbalanced  order  with  approximately  three 
weeks  interval  between  the  first  and  second  testing.  One  half  of 
the  subjects  were  given  the  simulated  tasks  first  while  the 
remainder  of  the  subjects  were  initially  given  the  actual  per¬ 
formance  task.  The  performance  version  of  the  test  was  given 
individually  in  a  room  separate  from  where  the  simulated  form  was 
administered.  One  observer  monitored  the  simulated  test  admin¬ 
istration  to  ascertain  that  test  procedures  were  correctly  followed 
by  all  examinees.  One  technical  observer  was  used  for  each  examinee 
in  administering  and  recording  performnace  on  the  actual  super¬ 
heterodyne  task.  Observers  were  former  Navy  technicians  with 
considerable  recent  experience  in  developing  and  grading  performance 
tests. 

Although  the  simulated  test  format  was  generally  familiar  to 
students  in  the  experimental  training  course,  one  form  (9B)  was 
administered  first  to  provide  immediate  practice  and  to  obscure 
the  relationship  between  the  common  simulated  and  actual  test 
casualty.  None  of  the  subjects  apparently  perceived  the  similarity 
of  the  casualty  used  in  the  two  test  versions,  separated  by  a 
three-week  interval.  This  lack  of  recognition  may  be  ascribed  to 
several  favorable  factors  in  the  experimental  situation.  The 
experimental  tests  were  administered  by  the  same  technical  personnel 
who  administered  regular  within-course  and  final  examinations  for 
the  training  course.  Additionally,  the  experimental  tests  were 
intermixed  with  regular  course  testings,  and  were  of  such  similar 
nature  as  to  be  indistinguishable  from  them,  since  simulated  and 
actual  troubleshooting  performance  constituted  a  large  portion  of 
the  course  work. 

4.  Test  Scoring 


Grading  of  performance  tests  is  typically  a  difficult  and 
uncertain  task,  and  the  present  measurements  were  no  exception. 
There  is  no  specified  method  for  objectively  scoring  the 
Van  Valkenburgh  forms,  and  the  actual  superheterodyne  performance 
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test  was  amenable  to  scoring  only  to  the  extent  that  any  complex 
performance  task  involving  real  equipment  and  casualties  may  be. 
The  scoring  procedures  employed  in  the  present  investigation  were 
determined,  to  a  great  extent,  by  experience  from  considerable 
previous  research  work  with  both  these  types  of  tests.  An  effort 
was  made  to  score  the  most  objective  factors  yielded  by  the 
Van  Valkenburgh  forms,  and  to  score  similar  aspects  on  the  actual 
troubleshooting  test  to  provide  a  basis  for  meaningful  performance 
comparisons.  To  this  end,  a  scoring  sheet  was  developed  for  the 
actual  task  which  paralleled  the  listing  and  identification  of 
components  and  check  points  given  in  the  simulated  version.  It 
was  expected  that  although  any  individual’s  absolute  scores  on 
the  simulated  task  might  be  considerably  different  from  those  on 
the  actual  task,  his  relative  performance  might  be  constant  enough 
to  yield  consistent  performance  score  rankings  for  subjects  across 
tests. 


Six  relatively  objective  scores  were  obtained  from  performance 
on  each  version  of  the  test:  (1)  Number  of  Steps,  (2)  Correct 
Steps,  (3)  Incorrect  Steps,  (4)  Total  Time,  (5)  Tester's  Rating, 

(6)  Parts  Replaced.  Certain  test  administration  procedures  were 
necessary  to  provide  this  scoring  information.  In  the  simulated 
test,  examinees  were  instructed  to  number  each  step  (erasure  to 
disclose  item  information)  in  sequence  from  their  first  "1"  to 
their  final  item.  In  the  actual  test,  the  monitor  recorded,  in 
order,  each  troubleshooting  step  the  examinee  made.  These  recorded 
step  sequences  were  later  transposed  to  the  scoring  sheet  parallel¬ 
ing  that  of  the  simulated  form. 


In  addition  to  noting  the  simple  step  sequence  (e.g.  signal 
inject  at  V2-8) ,  monitors  in  the  actual  testing  were  able  to 
record  detailed  descriptions  of  the  procedures  employed  by  the 
examinee  in  performing  each  step.  Thus  a  record  was  made  not  only 
of  what  was  done,  but  how  it  was  accomplished  (e.g.  power  level, 
frequency,  and  procedural  facility  employed  in  injecting  the  signal 
at  V2-8) .  No  such  relevant  performance  characteristics  were 
available  from  the  simulated  test  forms  since  it  is  explicitly 
assumed  in  that  version  that  "...all  checks  and  measurements  are 
made  with  the  proper  instruments,  correctly  used  and  interpreted." 
For  both  simulated  and  actual  tests  a  stopwatch  record  was 
maintained  of  the  total  elapsed  time  for  the  troubleshooting  per¬ 
formance  of  each  examinee,  and  ten  minute  intervals  were  indicated 
throughout  the  testing. 


The  score  for  Number  of  Steps,  therefore,  was  simply  the  total 
number  of  checks,  measurements,  or  part  replacements  recorded  on 
the  scoring  sheets  for  both  simulated  and  actual  tests.  The  Number 
of  Correct  Steps  was  derived  from  this  total,  and  required  the 
technical  judgements  of  the  test  observers  after  viewing  the  total 
recorded  test  performance.  "Correct"  steps  were  those  which  were 
judged,  within  the  total  approach  followed  by  the  individual  testee, 
to  be  logical  alternative  troubleshooting  checks  which  would  yield 
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information  relevant  to  locating  the  casualty.  "Incorrect"  steps 
were  those  judged  to  be  illogical,  redundant,  or  otherwise  not 
reasonable  alternative  steps  toward  casualty  isolation.  Despite 
the  subjective  element  of  these  two  scores,  it  was  found  that  the 
technical  judges  were  able  to  make  reliable  and  congruent  decisions 
based  on  the  relation  of  individual  steps  to  total  test  sequence. 

This  scoring  method  allows  equal  credit  for  any  of  a  number  of  minor 
variations  in  the  troubleshooting  sequence,  since  the  correctness 
of  each  step  depends  upon  how  well  it  fits  into  the  total  approach 
uniquely  employed  by  each  individual. 

The  score  for  Total  Time  was  simply  the  number  of  minutes 
required  by  each  examinee  to  finish  the  simulated  and  the  actual 
task.  Variations  in  troubleshooting  progress  for  each  subject 
within  the  recorded  ten  minute  intervals  did  not  yield  any  dis¬ 
criminating  grading  information  and  was  therefore  not  included  in 
the  final  score  analyses. 

The  Tester's  Rating  comprised  the  overall  proficiency  grade 
assigned  by  the  technical  observer.  This  score  is  more  meaning¬ 
fully  applicable  to  the  actual  performance  test  where  a  number  of 
relevant  observable  troubleshooting  skills  (use  of  test  equipment, 
safety  Procedures,  etc.)  may  give  a  more  comprehensive  indication 
of  overall  performance  proficiency.  The  Tester's  Rating  score  was 
assigned  on  the  basis  of  a  five  point  scale  which  incorporates 
descriptive  statements  of  the  typical  performance  characteristics 
exemplifying  each  of  the  proficiency  levels  from  1  "inadequate" 
to  5  "excellent." 

The  score  for  Parts  Replaced  was  the  total  number  of  items 
erased  in  this  category  on  the  simulated  test.  Incorrect  parts 
replacement  items,  when  erased,  yielded  the  information  "SR" 

(Symptom  remains)  indicating  that  further  steps  were  required. 

For  the  actual  performance  test  this  score  was  similarly  the 
number  of  components  which  were  replaced  in  an  attempt  to  repair 
the  casualty. 

5» 

C.  Results  and  Discussion 


The  results  of  this  experimental  comparison  of  simulated  and 
actual  tasks  are  primarily  intended  as  an  analysis  of  performance 
characteristics  evoked  by  different  test  modes  rather  than  as  a 
statistical  assessment  of  the  psychometric  properties  of  the  test 
instruments.  The  relatively  small  sample  size  and  the  absence  of 
unitary  test  scores  of  confirmed  reliability  preclude  the  presenta¬ 
tion  of  test  results  in  terms  of  total  or  part-test  score  validities. 
In  order  to  maximize  the  number  of  meaningful  performance  compari¬ 
sons,  however,  inter-test  correlations  were  obtained  for  five  of 
the  six  scored  performance  variables.  (The  range  of  scores  for 
Parts  Replaced  in  the  actual  test  was  too  limited  for  meaningful 
correlational  analyses.)  Certain  other,  less  statistically 
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specifiable  performance  results  are  also  included  as  relevant  to  the 
general  problem  of  proficiency  measurement. 

1.  Test  Scores 


A  comparison  of  the  scores  obtained  on  the  simulated  and  on 
the  actual  performance  tests,  in  terms  of  range  and  average,  is 
presented  in  Table  1.  The  simulated  practice  test  results  are 
included  in  this  and  subsequent  tables. 


TABLE  1 

Mean  and  Range  of  Scores  on  Simulated 
and  Actual  Tasks  (N=14) 


Scores 

Tests 

Actual 

Simulated 

Simulated  (Practice) 

No.  of  Steps 

L 

7 

6 

6 

H 

31 

38 

31 

r 

13.9 

17.6 

14.6 

Correct  Steps 

L 

5 

5 

3 

H 

15 

21 

15 

T 

8.6 

9.9 

7.5 

Incorrect  Steps 

L 

0 

1 

0 

H 

25 

32 

24 

X 

5.3 

7.7 

7.1 

Total  Time 

L 

10 

5 

5 

H 

70 

32 

43 

X 

29.1 

18.6 

19.6 

Tester's  Rating 

L 

1 

1 

1 

H 

4 

4 

5 

x 

3.1 

2.9 

2.8 

Parts  Replaced 

L 

1 

1 

1 

H 

2 

8 

5 

x 

1.6 

3.1 

2.3 

The  comparisons  of  Table  1  indicate  a  fair  degree  of  similarity 
between  performance  scores  on  the  two  simulated  tests,  with  the 
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practice  form  requiring  slightly  greater  average  time  for  comple¬ 
tion,  possibly  as  a  result  of  order  of  administration.  As  expected, 
the  actual  performance  test  required  considerably  more  time  to 
complete,  on  the  average,  than  either  simulated  test.  The  most 
notable  difference  apparent  from  the  table  is  the  discrepancy 
between  simulated  and  actual  tasks  in  the  number  of  parts  replaced. 
No  examinee  replaced  more  than  two  parts  in  the  actual  task.  (One 
replacement  was  necessary  to  repair  the  casualty) .  In  the  simu¬ 
lated  version,  however,  as  many  as  eight  parts  were  replaced.  It 
is  evident  that  this  difference  was  largely  due  to  the  fact  that 
parts  replacement  in  the  simulated  task  required  only  a  simple 
erasure,  whereas  in  the  actual  task  considerably  more  effort  was 
involved.  This  consideration  probably  affected  all  actual  perform¬ 
ance  steps  (checks  and  measurements)  as  indicated  by  the  slightly 
lower  average  number  of  total  steps  for  the  actual  test  sequences. 

2.  Inter- test  Correlations 


The  rank  order  inter-test  correlations  for  the  five  scored 
performance  variables  are  presented  in  Table  2. 


TABLE  2 

Rank  Order  Inter-test  Correlations  for 
Five  Performance  Variables  (N=14) 


Test  Score 

Tests 

Simulated 
x  Actual 

Simulated 
(Practice) 
x  Actual 

Simulated  x 
Simulated 
(Practice) 

No.  of  Steps 

-.32 

.40 

-.12 

No.  of  Correct  Steps 

.14 

.69 

-.10 

No.  of  Incorrect  Steps 

-.19 

-.07 

-.13 

Total  Time 

-.50 

.30 

-.26 

Tester's  Rating 

-.35 

-.06 

-.16 

The  correlational  comparisons  of  Table  2  reveal  a  number  of 
striking  features  concerning  the  relationship  between  the  perform¬ 
ance  variables  on  the  simulated  and  on  the  actual  tasks. 


A  predominant  feature  is  the  lack  of  positive  relationship 
between  performance  scores  on  the  simulated  test  and  scores  on  the 
actual  task  which  involved  an  identical  troubleshooting  problem 
and  associated  job  aids.  Indeed,  four  of  the  five  scored  variables 
are  negatively  related,  with  the  highest  of  these  (-.55)  based  upon 
the  objectively  determined  Total  Time. 

A  secondary  consideration  is  the  fact  that  the  scores  on  the 
two  simulated  tests,  the  first  administered  for  practice  purposes, 
bear  no  relationship  to  each  other  despite  the  virtual  equivalence 
of  the  two  tasks  in  all  aspects  except  the  specific  problem  casu¬ 
alty.  Again,  the  comparison  of  Total  Time  yielded  the  poorest 
coefficient  for  positive  prediction  (-.26). 

Tester's  Rating,  which  was  designed  to  provide  the  best  overall 
summary  of  task  proficiency,  is  negatively  correlated  for  all  three 
inter-test  comparisons . 

The  highest  inter-test  correlation  involves  Number  of  Correct 
Steps  on  the  actual  test  and  on  the  simulated  test  used  for 
practice.  Since  this  simulated  test  involved  a  different  casualty 
from  the  actual  test,  and  since  scores  for  Number  of  Incorrect 
Steps,  and  Tester's  Rating  are  both  uncorrelated,  it  is  difficult 
to  view  this  single  coefficient  as  representing  a  realistic 
relationship  of  practical  predictive  utility. 

No  other  individual  coefficients  in  this  table  command  atten¬ 
tion.  It  may,  in  fact,  be  noted  that  of  the  fifteen  inter-test 
comparisons,  only  four  yielded  positive  correlations.  Three  of 
these  four  involve  comparisons  between  the  practice  test  and  the 
actual  performance  test,  where  meaningful  relationships  would 
logically  be  least  expected. 

3.  Test  Evaluation 


In  view  of  the  lack  of  relationship  between  simulated  and  actual 
performance  variables,  a  subsidiary  attempt  was  made  to  estimate 
the  practical  validity  of  the  tests  involved,  in  terms  of  an  out¬ 
side  measure  of  actual  troubleshooting  performance .  For  this 
purpose,  correlations  were  computed  between  Tester's  Ratings  and 
the  performance  achievement  scores  of  examinees  for  the  Equipment 
phase  of  the  experimental  training  course.  This  performance 
achievement  score  was  based  on  a  total  of  approximately  thirty 
performance  tests  taken  during,  and  at  the  end  of  the  training 
course.  Table  3  presents  the  resultant  rank  order  coefficients. 
These  obtained  correlations  tend  to  support  the  assumption  that 
the  actual  superheterodyne  test,  developed  for  this  study,  is  a 
better  job  sample  of  real  performance  tasks  than  are  the  simulated 
performance  tasks. 
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TABLE  3 


Rank  Order  Correlation  of  Tester's  Ratings 
With  X-ET  Course  Performance  Grade 


Test  Rho 

Actual  Performance  .55 
Simulated  Performance  -.36 
Simulated  Performance  (Practice)  .00 


4.  Performance  Characteristics 


There  are  a  number  of  important  non-statistical  comparisons 
between  the  simulated  and  actual  performance,  which  involve  aspects 
other  than  the  six  scored  variables.  These  performance  comparisons 
are  based  largely  on  the  test  observations  and  interview  informa¬ 
tion  provided  by  the  technical  monitors.  From  these  comparisons 
it  is  apparent  that  there  were  many  differences  in  task  approach, 
strategy,  and  technique  which  were  directly  associated  with  the 
test  medium.  These  performance  differences  probably  contribute 
to  the  negative  correlations  obtained  between  simulated  and  actual 
scores . 


In  the  simulated  performance  situation  all  checks  and  measure¬ 
ments  are  obtained  by  simple  erasure,  and  results  given  are 
invariably  accurate.  In  actual  performance,  checks  and  measurements 
require  considerably  more  effort,  and  the  accuracy  of  the  readings 
depends  upon  how  correctly  the  test  equipment  is  used.  Thus,  in 
taking  the  actual  performance  test,  examinees  often  repeated  the 
same  check  or  measure  because  they  were  not  sure  of  the  accuracy 
of  their  findings.  Test  protocols  show  as  many  as  eight  similar 
checks  made  by  testees  at  a  single  test  point  (e.g.  plate  of  V-2) , 
even  though  a  single  correct  check  would  be  sufficient  to  provide 
all  available  information  at  that  point.  Dubious  measurements 
tended  to  affect  the  entire  troubleshooting  sequence.  Reliance 
upon  an  incorrect  reading,  for  example,  could  lead  examinees  to  a 
false  casualty  assumption.  Conversely,  uncertainty  over  a  correct 
reading  sometimes  caused  students  to  persist  in  repeating  an 
unproductive  line  of  troubleshooting  strategy. 


There  were  also  substantial  differences  in  the  number  of  parts 
replaced  under  the  two  test  situations,  as  previously  noted.  In 
the  actual  task,  students  were  reluctant  to  unsolder  or  disconnect 


components  from  the  chassis,  but  in  the  simulated  task,  where  parts 
replacement  required  virtually  no  effort,  students  too  often 
resorted  to  parts  replacement  in  an  effort  to  solve  the  problem. 

The  test  format  also  affected  overall  troubleshooting  strategy 
to  some  extent,  as  indicated  by  detailed  comparison  of  the  proce¬ 
dural  sequences  used  by  the  same  testee  on  the  simulated  and  on 
the  actual  task.  Only  one  of  the  fourteen  testees  even  made  the 
same  initial  troubleshooting  check  on  both  his  actual  and  simulated 
test. 

In  the  actual  performance  task  students  frequently  made  basic 
errors  in  test  equipment  usage,  logic,  and  safety  procedures,  which 
were  not  revealed  by  the  simulated  task.  Examples  of  commonly 
observed  errors  include:  touching  high  voltage  points  when  taking 
B+  measurements,  injecting  IF  into  the  antenna,  taking  resistance 
readings  with  the  power  on,  setting  signal  generator  input  levels 
too  low,  neglecting  to  zero  meters,  improper  setting  of  tube  tester 
controls. 


D .  Summary  and  Conclusions 


The  critical  need  for  performance  proficiency  measures  has 
engendered  the  hope  that  simulated  performance  tests  may  be 
employed  to  provide  reasonably  valid  estimates  of  actual  performance 
ability.  This  investigation  provided  an  opportunity  to  compare  the 
performance  of  subjects  on  carefully  devised  simulated  and  actual 
troubleshooting  performance  tasks  involving  identical  casualties 
and  job  aids. 

The  summary  results  of  this  experimental  comparison  follow: 

(1)  There  were  no  significant  positive  correlations  between 
the  simulated  performance  test  and  the  actual  performance  test  on 
any  of  the  five  scored  performance  variables.  Four  of  the  five 
inter-test  score  comparisons,  in  fact,  yielded  negative  correlations. 

(2)  There  were  observable  differences  in  specific  performance 
procedures  and  in  overall  troubleshooting  strategy  attributable 
to  the  differences  in  test  mode, 

(3)  Many  of  the  troubleshooting  ''enabling'1  skills  which 
seriously  affect  performance  in  the  actual  task  situation  were 
not  measurable  in  the  simulated  task  situation. 

(4)  There  were  no  positive  correlations  between  the  performance 
scores  of  the  two  simulated  tasks,  one  of  which  was  administered 
for  practice  purposes. 

(5)  The  actual  performance  test  developed  for  this  investiga¬ 
tion  had  considerable  face  validity  as  a  job  sample  of  trouble¬ 
shooting  tasks,  and  was  found  to  correlate  significantly  with  an 
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outside  performance  criterion.  Neither  of  the  simulated  tasks 
correlated  positively  with  this  performance  criterion. 

In  summary,  the  simulated  performance  measure  employed  in  this 
investigation  did  not  provide  a  valid  estimate  of  proficiency  on  the 
identical  problem  requiring  actual  performance.  Negative  inter-test 
correlations  indicate  that  simulated  test  scores  would  actually  be 
misleading  in  terms  of  estimating  actual  performance  scores. 

The  restricted  generality  of  the  present  study  does  not  admit 
the  conclusion  that  simulated  tests  are  necessarily  without  value. 
Other  experimental  contexts  might  yield  more  positive  results,  and 
simulated  performance  tasks  may  serve  an  educative  and  practice 
function  in  training  apart  from  the  assessment  role.  Improvement 
of  the  simulated  test  by  modification  of  the  test  directions,  the 
scoring  methods,  and  the  task  procedures,  might  make  the  simulated 
measure  a  more  realistic  approximation  of  the  actual  performance 
task.  The  evidence  does,  however,  strongly  suggest  caution  in 
assuming  that  any  simulated  performance  measure,  even  when  it 
possess  considerable  common  identity  to  the  actual  task,  can  pro¬ 
vide  a  valid  estimate  of  actual  performance  proficiency.  In  the 
development  of  Navy-wide  systems  for  proficiency  evaluation, 
simulated  performance  measures  should  be  empirically  validated 
against  real  performance  criteria,  before  they  are  accepted  for 
incorporation  into  the  total  assessment  system. 
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